Hydroviz Test Suite¶
NOTE: there is an hourly limit to the number of calls we can make to the USGS water data API! If you see errors fetching observed data, this is likely the reason why.
NOTE: the y scale is plotted using symlog so that both small and large flows can be seen on the same plot. Flows between 0 and 10^0 (i.e. 1) are plotted linearly, flows above 10^0 are plotted logarithmically. Even so, some horizontal line artifacts may appear at 10^0 (i.e, flow values of 1) in data where flow values have been rounded to whole numbers. This is apparent in some observed data for small streams with low flows, and is not usually present in the modeled data.
Statistics calculated¶
Normalized Nash-Sutcliffe Efficiency (NNSE): this is a skill score between 0 and 1. A score of 1 indicates that the model predicts flow perfectly. A score of 0.5 indicates that the model has the same flow prediction skill as using the mean of observed values. Scores below 0.5 indicate that mean of observed values is a better predictor than the model. The limitations of using this score to compare skill between different streams are made clear in this paper. These scores should not be used to compare model performance between different streams, but can be used to compare model performance for the same stream. For example, we can compare NNSE values between landcover types (dyanmic vs static) and model types (baseline vs ensemble) for the same stream to see which model type had higher skill. Also note that when streamflow is not highly variable, the mean of the observations may in fact be a good predictor and in these cases the NNSE scores may be misleading. For example, imagine a stream with a very flat hydrograph where the mean of observations is very close to the streamflow line. In this case, assessing model skill against the observed mean is inappropriate, since the observed mean is actually a skilled predictor.
Normalized Mean Absolute Error (NMAE): this is the mean absolute error divided by the mean of observations. NMAE measures the typical magnitude of model error relative to a typical flow condition. By normalizing with the mean observed discharge, it answers: “On an average day, how large is the model’s error compared to the average streamflow?” We can use this value to assess overall agreement in typical conditions and performance across the bulk of the hydrograph. It is less sensitive to outliers than RMSE-based metrics, and we can safely compare this value between different stream segments. In general, NMAE < 0.3 means errors are small relative to typical flow; NMAE 0.3–0.6 means moderate but acceptable error, and NMAE 0.6–1.0 means errors comparable to mean flow (marginal performance). NMAE > 1.0 means typical error exceeds typical flow (poor performance). In nearly flat hydrographs, NMAE functions mainly as a normalized bias metric, and high values indicate that the modeled mean flow differs substantially from the observed mean, not that the model fails to capture variability or timing.
Normalized Root Mean Square Error (NRMSE): this is the root mean square error divided by the range of observations. NRMSE measures how large model errors are relative to the full dynamic range of observed flows, with strong emphasis on peaks and timing errors. It answers: “How large are the largest (squared) errors compared to the overall variability of the stream?” This metric has sensitivity to high flows and missed peaks, ability to capture flow variability, and gets at timing and magnitude errors during events. In general, NRMSE < 0.2 means errors are small relative to the system’s variability, NRMSE 0.2–0.4 means reasonable reproduction of flow range, and NRMSE 0.4–1.0 means substantial peak or timing errors. NRMSE > 1.0 means model errors exceed the observed flow range (very poor performance). When the hydrograph is nearly flat, NRMSE (normalized by the observed range) becomes unstable and largely reflects sensitivity to small absolute errors relative to a very small flow range, so high values indicate a scale/normalization issue rather than meaningful failure to reproduce flow dynamics.
How to use this notebook:¶
Start a local instance of the API, then execute the run_test_suite() function using the cells below. Using type "baseline" will plot the Maurer baseline and compute statistics vs observed USGS streamflow. Using type "ensemble" will plot the 13 model ensemble and compute statistics vs observed USGS streamflow
In [ ]:
from test_functions import *
import json
with open("test_streams.json", "r") as file:
test_streams = json.load(file)
In [ ]:
run_test_suite(test_streams, type="baseline")
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------- Warning: All modeled values are zero for one of the landcover types. Returning NaN for all statistics.
-----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------- Problem getting observed data for stream ID: 6260. Skipping.
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------
Stream ID: 50576 | Region: Marine West Coast Forest | Subregion: Coast Range
Gauge ID: USGS-12040500 | Gauge Name: QUEETS RIVER NEAR CLEARWATER, WA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.9317, NMAE: 0.1142, NRMSE: 0.0615
Landcover Type: Static
NNSE: 0.9315, NMAE: 0.1122, NRMSE: 0.0613
--------------------------------------------------------------------------------
Stream ID: 50920 | Region: Marine West Coast Forest | Subregion: Coast Range
Gauge ID: USGS-14307620 | Gauge Name: SIUSLAW RIVER NEAR MAPLETON, OR | Data Completeness: 75.8%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.9251, NMAE: 0.1822, NRMSE: 0.0821
Landcover Type: Static
NNSE: 0.9241, NMAE: 0.1847, NRMSE: 0.0816
--------------------------------------------------------------------------------
Stream ID: 52371 | Region: Western Cordillera | Subregion: North Cascades
Gauge ID: USGS-12175500 | Gauge Name: THUNDER CREEK NEAR NEWHALEM, WA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.8422, NMAE: 0.1804, NRMSE: 0.1155
Landcover Type: Static
NNSE: 0.8184, NMAE: 0.1916, NRMSE: 0.1241
--------------------------------------------------------------------------------
Stream ID: 52916 | Region: Western Cordillera | Subregion: North Cascades
Gauge ID: USGS-12447383 | Gauge Name: METHOW RIVER ABOVE GOAT CREEK NEAR MAZAMA, WA | Data Completeness: 46.4%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.9684, NMAE: 0.2034, NRMSE: 0.0508
Landcover Type: Static
NNSE: 0.9453, NMAE: 0.2532, NRMSE: 0.0683
--------------------------------------------------------------------------------
Stream ID: 50291 | Region: Western Cordillera | Subregion: Cascades
Gauge ID: USGS-14179000 | Gauge Name: BREITENBUSH R ABV FRENCH CR NR DETROIT, OR. | Data Completeness: 62.1%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.5810, NMAE: 0.2239, NRMSE: 0.1382
Landcover Type: Static
NNSE: 0.6572, NMAE: 0.2026, NRMSE: 0.1188
--------------------------------------------------------------------------------
Stream ID: 50262 | Region: Western Cordillera | Subregion: Cascades
Gauge ID: USGS-14161500 | Gauge Name: LOOKOUT CREEK NEAR BLUE RIVER, OR | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.9029, NMAE: 0.1569, NRMSE: 0.0797
Landcover Type: Static
NNSE: 0.8901, NMAE: 0.1610, NRMSE: 0.0771
--------------------------------------------------------------------------------
Stream ID: 54594 | Region: Mediterranean California | Subregion: California Coastal Sage, Chaparral, and Oak Woodlands
Gauge ID: USGS-11463170 | Gauge Name: BIG SULPHUR C A G RESORT NR CLOVERDALE CA | Data Completeness: 86.2%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.1219, NMAE: 0.6506, NRMSE: 0.0808
Landcover Type: Static
NNSE: 0.1544, NMAE: 0.6359, NRMSE: 0.0787
--------------------------------------------------------------------------------
Stream ID: 55353 | Region: Mediterranean California | Subregion: California Coastal Sage, Chaparral, and Oak Woodlands
Gauge ID: USGS-11113500 | Gauge Name: SANTA PAULA C NR SANTA PAULA | Data Completeness: 96.5%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.7940, NMAE: 0.4418, NRMSE: 0.0738
Landcover Type: Static
NNSE: 0.8151, NMAE: 0.3905, NRMSE: 0.0729
--------------------------------------------------------------------------------
Stream ID: 47244 | Region: Cold Deserts | Subregion: Central Basin and Range
Gauge ID: USGS-10322500 | Gauge Name: HUMBOLDT RV AT PALISADE, NV | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.5169, NMAE: 1.7844, NRMSE: 0.6601
Landcover Type: Static
NNSE: 0.5429, NMAE: 1.5114, NRMSE: 0.5367
--------------------------------------------------------------------------------
Stream ID: 47362 | Region: Cold Deserts | Subregion: Central Basin and Range
Gauge ID: USGS-10329500 | Gauge Name: MARTIN CK NR PARADISE VALLEY, NV | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.6923, NMAE: 0.4021, NRMSE: 0.1414
Landcover Type: Static
NNSE: 0.6268, NMAE: 0.4118, NRMSE: 0.1489
--------------------------------------------------------------------------------
Stream ID: 43861 | Region: Cold Deserts | Subregion: Colorado Plateaus
Gauge ID: USGS-09333500 | Gauge Name: DIRTY DEVIL R AB POISON SP WSH NR HANKSVILLE UT | Data Completeness: 74.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.2989, NMAE: 4.1317, NRMSE: 1.0883
Landcover Type: Static
NNSE: 0.3627, NMAE: 2.1585, NRMSE: 0.6338
--------------------------------------------------------------------------------
Stream ID: 43914 | Region: Cold Deserts | Subregion: Colorado Plateaus
Gauge ID: USGS-09306500 | Gauge Name: WHITE RIVER NEAR WATSON, UTAH | Data Completeness: 79.3%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.5108, NMAE: 0.9880, NRMSE: 0.4442
Landcover Type: Static
NNSE: 0.4919, NMAE: 0.9555, NRMSE: 0.4249
--------------------------------------------------------------------------------
Stream ID: 56347 | Region: Warm Deserts | Subregion: Mojave Basin and Range
Gauge ID: USGS-10261500 | Gauge Name: MOJAVE R A LO NARROWS NR VICTORVILLE CA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.7574, NMAE: 0.6112, NRMSE: 0.0977
Landcover Type: Static
NNSE: 0.7504, NMAE: 0.6518, NRMSE: 0.1014
--------------------------------------------------------------------------------
Stream ID: 55769 | Region: Warm Deserts | Subregion: Mojave Basin and Range
Gauge ID: USGS-10251300 | Gauge Name: AMARGOSA RV AT TECOPA, CA | Data Completeness: 47.2%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.1063, NMAE: 76.9340, NRMSE: 2.4096
Landcover Type: Static
NNSE: 0.0921, NMAE: 74.7574, NRMSE: 2.3183
--------------------------------------------------------------------------------
Stream ID: 44935 | Region: Warm Deserts | Subregion: Sonoran Desert
Gauge ID: USGS-09484000 | Gauge Name: SABINO CREEK NEAR TUCSON, AZ | Data Completeness: 62.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.0009, NMAE: 0.9808, NRMSE: 0.1658
Landcover Type: Static
NNSE: 0.0009, NMAE: 0.9829, NRMSE: 0.1658
--------------------------------------------------------------------------------
Stream ID: 56271 | Region: Warm Deserts | Subregion: Sonoran Desert
Gauge ID: USGS-10259200 | Gauge Name: DEEP C NR PALM DESERT CA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.5733, NMAE: 1.9041, NRMSE: 0.2677
Landcover Type: Static
NNSE: 0.6052, NMAE: 1.2449, NRMSE: 0.2480
--------------------------------------------------------------------------------
Stream ID: 24197 | Region: Great Plains | Subregion: High Plains
Gauge ID: USGS-06710150 | Gauge Name: BIG DRY CREEK BELOW C-470 AT HIGHLANDS RANCH, CO | Data Completeness: 6.9%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.0073, NMAE: 0.7167, NRMSE: 0.0934
Landcover Type: Static
NNSE: 0.0230, NMAE: 0.7012, NRMSE: 0.0900
--------------------------------------------------------------------------------
Stream ID: 27446 | Region: Great Plains | Subregion: High Plains
Gauge ID: USGS-06762500 | Gauge Name: Lodgepole Creek at Bushnell, Nebr. | Data Completeness: 62.1%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.3083, NMAE: 8.2720, NRMSE: 0.9180
Landcover Type: Static
NNSE: 0.2523, NMAE: 7.5367, NRMSE: 0.7649
--------------------------------------------------------------------------------
Stream ID: 27801 | Region: Great Plains | Subregion: Central Great Plains
Gauge ID: USGS-06872500 | Gauge Name: NF SOLOMON R AT PORTIS, KS | Data Completeness: 99.6%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.4901, NMAE: 0.4340, NRMSE: 0.1429
Landcover Type: Static
NNSE: 0.3959, NMAE: 0.4851, NRMSE: 0.1535
--------------------------------------------------------------------------------
Stream ID: 28087 | Region: Great Plains | Subregion: Central Great Plains
Gauge ID: USGS-06853800 | Gauge Name: WHITE ROCK C NR BURR OAK, KS | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.6626, NMAE: 0.4491, NRMSE: 0.1196
Landcover Type: Static
NNSE: 0.6350, NMAE: 0.4170, NRMSE: 0.1144
--------------------------------------------------------------------------------
Stream ID: 31380 | Region: Great Plains | Subregion: Northwestern Great Plains
Gauge ID: USGS-06130500 | Gauge Name: Musselshell River at Mosby MT | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.1868, NMAE: 2.2492, NRMSE: 0.7153
Landcover Type: Static
NNSE: 0.1973, NMAE: 2.3918, NRMSE: 0.7659
--------------------------------------------------------------------------------
Stream ID: 29015 | Region: Great Plains | Subregion: Northwestern Great Plains
Gauge ID: USGS-06214500 | Gauge Name: Yellowstone River at Billings MT | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.9479, NMAE: 0.1663, NRMSE: 0.0673
Landcover Type: Static
NNSE: 0.9524, NMAE: 0.1627, NRMSE: 0.0656
--------------------------------------------------------------------------------
Stream ID: 17871 | Region: Great Plains | Subregion: Western Corn Belt Plains
Gauge ID: USGS-05316500 | Gauge Name: REDWOOD RIVER NEAR REDWOOD FALLS, MN | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.2332, NMAE: 0.5348, NRMSE: 0.2154
Landcover Type: Static
NNSE: 0.2500, NMAE: 0.5281, NRMSE: 0.2086
--------------------------------------------------------------------------------
Stream ID: 24828 | Region: Great Plains | Subregion: Western Corn Belt Plains
Gauge ID: USGS-06819185 | Gauge Name: East Fork 102 River at Bedford, IA | Data Completeness: 75.9%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.0536, NMAE: 0.7821, NRMSE: 0.1664
Landcover Type: Static
NNSE: 0.0440, NMAE: 0.8183, NRMSE: 0.1715
--------------------------------------------------------------------------------
Stream ID: 4521 | Region: Eastern Temperate Forests | Subregion: Piedmont
Gauge ID: USGS-02331600 | Gauge Name: CHATTAHOOCHEE RIVER NEAR CORNELIA, GA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.8371, NMAE: 0.1244, NRMSE: 0.0963
Landcover Type: Static
NNSE: 0.8644, NMAE: 0.1176, NRMSE: 0.0989
--------------------------------------------------------------------------------
Stream ID: 4889 | Region: Eastern Temperate Forests | Subregion: Piedmont
Gauge ID: USGS-02337000 | Gauge Name: SWEETWATER CREEK NEAR AUSTELL, GA | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.7635, NMAE: 0.1984, NRMSE: 0.0921
Landcover Type: Static
NNSE: 0.8541, NMAE: 0.1592, NRMSE: 0.0751
--------------------------------------------------------------------------------
Stream ID: 4847 | Region: Eastern Temperate Forests | Subregion: Southeastern Plains
Gauge ID: USGS-02374500 | Gauge Name: MURDER CREEK NEAR EVERGREEN AL | Data Completeness: 100.0%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.7662, NMAE: 0.1717, NRMSE: 0.0895
Landcover Type: Static
NNSE: 0.7660, NMAE: 0.1766, NRMSE: 0.0915
--------------------------------------------------------------------------------
Stream ID: 9245 | Region: Eastern Temperate Forests | Subregion: Southeastern Plains
Gauge ID: USGS-02427250 | Gauge Name: PINE BARREN CREEK NEAR SNOW HILL, AL. | Data Completeness: 55.2%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.7170, NMAE: 0.2674, NRMSE: 0.0780
Landcover Type: Static
NNSE: 0.7117, NMAE: 0.2750, NRMSE: 0.0784
--------------------------------------------------------------------------------
Stream ID: 34617 | Region: Eastern Temperate Forests | Subregion: South Central Plains
Gauge ID: USGS-07344450 | Gauge Name: Paw Paw Bayou near Greenwood, LA | Data Completeness: 36.2%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
NNSE: 0.4075, NMAE: 0.6352, NRMSE: 0.0830
Landcover Type: Static
NNSE: 0.3993, NMAE: 0.6304, NRMSE: 0.0832
--------------------------------------------------------------------------------
Stream ID: 35668 | Region: Eastern Temperate Forests | Subregion: South Central Plains
Gauge ID: USGS-07351500 | Gauge Name: Cypress Bayou near Keithville, LA | Data Completeness: 75.9%
Model: Maurer | Scenario: historical | Era: 1976-2005
Landcover Type: Dynamic
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[2], line 1 ----> 1 run_test_suite(test_streams, type="baseline") File ~/hydroviz/data/tests/test_functions.py:91, in run_test_suite(test_streams, type) 88 # break 90 print("\n\n\n") ---> 91 print_stats_summary(stats_list) 93 return None File ~/hydroviz/data/tests/test_functions.py:523, in print_stats_summary(stats_list) 520 for landcover in ["dynamic", "static"]: 521 print(f" Landcover Type: {landcover.capitalize()}") 522 print( --> 523 f" NNSE: {stats[landcover]['NNSE']:.4f}, NMAE: {stats[landcover]['NMAE']:.4f}, NRMSE: {stats[landcover]['NRMSE']:.4f}" 524 ) 525 print("-" * 80) 527 return None ValueError: Unknown format code 'f' for object of type 'str'
In [ ]: